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Abstract 


Designing instruments for children and youth that result in reliable and valid data requires 
consideration beyond calculating grade-level equivalence of the text. Very little methodological 
research has been conducted on the survey response processes of children and youth and there 
are no comprehensive guidelines informing instrument development for this population. This 
paper reviews and integrates theories from the fields of cognitive, developmental, and 
educational psychology, as well as survey methodology, to consider how children’s cognitive, 
language/reading, and social/moral development impacts their progression through the four 
stages (comprehension, retrieval, judgment, and reporting) of the survey response model. Based 
on this review, a set of theoretically-based recommendations is proposed for designing or 
adapting instruments for children and youth that are 9 to 18 years old. These recommendations 
aim to minimize developmentally-related measurement errors within each stage of the response 


process. 


Introduction 


When conducting research on children and youth, the reliability and validity of the data can be 
questionable. Although proxy respondents (e.g., parents, caregivers, teachers) have traditionally 
been used for very young children, over time children become more able to answer questions 
about themselves and provide information that their parents do not know (Scott 1997). 
Unfortunately, there is little empirical evidence to guide decisions for collecting data from 
children and youth directly and there are no comprehensive guidelines informing instrument 
development for a young target population. This leaves researchers in the precarious position of 
collecting data with instruments that may not be appropriate for children and youth. 


Methodological research with children and youth is scarce for several reasons. In many countries 
across the world, minors are a protected class of research participants. In the United States, 
research with minors may involve additional scrutiny by institutional review boards, restricted 
access to participants, and parental consent. When sampling children through child care centers 
and schools, it may also be necessary to obtain consent from districts, schools, and/or teachers. 
These factors make research with children more challenging. 


Survey respondents are assumed to have basic abilities in three domains: cognition, 
language/reading, and social/moral systems. Yet, this assumption may not hold with young 
respondents whose abilities are still developing. Although previous researchers have discussed 
how cognitive development affects survey research with children and youth (e.g., Borgers, de 
Leeuw, and Hox 2000), few practical recommendations are available to guide instrument design 
with young populations. In terms of language and literacy, government surveys for adults are 
designed to meet a 5th grade reading level (Biemer and Lyberg 2003). Although social and moral 
skills provide context for responses, no previous research could be found on how lower social 
skills affect the response process in adults, much less in children or youth. 


Several disciplines offer literature and theories that can inform instrument design with children 
and youth, including cognitive psychology, developmental psychology, educational psychology, 
and survey methodology. These literatures and theories should be integrated to better understand 
how underdeveloped skills in youth may result in survey response errors, and how to reduce such 
errors. To integrate these resources, this paper has four sections. First, the survey response model 
by Tourangeau, Rips, and Rasinki (2000) is described, which serves as a framework to lend 
context to further sections of the article. Second, an overview of the development of cognitive, 
language and reading, and social and moral abilities is provided with detail as to how these skills 
interact. Next is a discussion of how underdeveloped skills in the three areas should theoretically 
affect survey responses. Finally, a set of theoretically-based recommendations is proposed for 
designing or adapting instruments for children and youth that are 9-18 years old to minimize 
developmentally-related measurement errors. For the purposes of this paper, an instrument refers 
to a survey, questionnaire, measure, or scale measuring beliefs, attitudes, or behaviors. 


The Survey Response Model 


The most commonly accepted theory of measurement error is based on the survey response 
model of Tourangeau and colleagues (2000). This model states that when answering survey 
questions, people progress through four cognitive stages: comprehension, retrieval, judgment, 
and reporting. If all goes well, the true answer matches the reported answer. Otherwise, 
measurement error results. 


In the first stage (comprehension), respondents perceive the question and attempt to understand 
its meaning. Measurement error can occur during this stage if the question contains unfamiliar 
language or content and respondents proceed with an incomplete understanding of what they are 
being asked. In the second stage (retrieval), respondents attempt to recall the relevant 
information. Measurement errors during retrieval occur when respondents are unable to 
remember or retrieve the wrong information. In the third stage (judgement), the respondent 
performs additional actions on retrieved information such as judging the adequacy of retrieved 
information, evaluating its accuracy, integrating it into a summary answer, or performing 
calculations. Measurement errors occur during judgment if respondents use a strategy to 
compensate for incorrect or incomplete retrieval, such as guessing, estimating, or relying on 
schemas and stereotypes. In the final stage (reporting), respondents determine how to report their 
answers. Measurement errors occur during reporting when an acceptable answer is not provided 
or the answer is edited to be socially acceptable. 


Ideally, respondents complete all four stages carefully and completely, and seek to address 
problems. For example, respondents might seek clarification, reattempt retrieval, or use outside 
information to supplement retrieval. Less attentive or motivated respondents will be more 
accepting of errors or even skip one or more steps, resulting in poorer data quality and response 
fidelity. As the next section will show, youths’ ability to complete these stages is dependent on 
their level of development in three domains: cognition, language/reading, and social/moral 
systems. 


Developmental Literature 


To inform developmentally-appropriate instrument design, this section provides a brief review of 
relevant literature from three key domains: cognition, language/reading, and 

social/moral systems. From each area, the key theories are summarized with associated ages or 
grade spans where possible. Readers are referred to the cited sources for more comprehensive 
explorations of these models. 


Cognitive Development 


The stages of cognitive development have implications for collecting data from children and 
youth because response quality is undoubtedly influenced by the developmental capacity to 
respond (Borgers, de Leeuw, and Hox 2000). The most widely-known and accepted model of 
cognitive development is Piaget’s stages of development (Feldman 2004; Piaget 1948). Piaget’s 
model of cognitive development consists of four sequential stages (see Figure 1). For the purpose 
of this paper, the focus is on the two latter stages. 


Figure 1. Piaget’s (1948) stages of cognitive development 


The concrete operational stage is most common for children ages 6 to 12 years old. In this stage, 
children are characterized by their understanding of the surrounding world in dichotomous, 
“black and white” terms. They begin to analyze things using more “adult-like” logic, although 
grounded in concrete systems (Piaget 1954). Children in this stage demonstrate the ability to 
follow logical operations in their mental processes, though they may have difficulties grasping 
concepts such as justice or fairness (Inhelder and Piaget 1964). The final stage, formal 
operational, begins between the ages of 12 and 18. In this stage, youth begin to use abstract 
reasoning to explore hypothetical scenarios that are unrelated to their own personal experiences 
(Inhelder and Piaget 1958). Youth entering this stage are better able to elaborate on their 
experiences and connect them to other situations. 


Concurrent to this cognitive skill development, children also incrementally improve their own 
understanding of these skills (Pillow 2008). Working memory capacity—the ability to hold 
information in one’s head while completing a task (e.g., Cowan 2008)—increases with age until 
it is fully developed around the age of 12 (Demetriou, Mouyi, and Spanoudis 2008). 
Additionally, as youth proceed through adolescence, they improve considerably in their ability to 
utilize metacognitive strategies for cognitive self-regulation (Huizinga, Dolan, and van der 
Molen 2006). 


Language and Reading Development 


Survey instruments consist of questions or items for respondents to answer by themselves or with 
the aid of an interviewer. As such, the use of instruments assumes that respondents have 
minimally sufficient language skills to engage in the question-answer process. Because 
instruments are largely designed to be completed independently, a certain level of reading 
proficiency is also assumed. For this, theory on language and reading development are relevant. 


Language development theory. Language consists of socially-shared rules for the meaning of 
words, conjugating new words, combining words into meaningful phrases, and situational 
appropriateness (American Speech-Language-Hearing Association 2016). Language acquisition 
is a meaning-making process that progresses throughout the lifespan (Halliday 1993). Children 
unconsciously interpret patterns throughout their language experiences and develop mental 
representations that can be applied in novel contexts (Gee 1994). Thus, language development is 
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a process of generalizing “rules” learned in one context to situations beyond what can be 
explicitly taught (Gee 1994). This development follows a predictable pattern, moving from a 
reliance on the literal meaning of words to applying an increasingly complex set of implicit, 
irregular rules to interpret the true meaning of a statement (Chomsky 1972). Factors known to 
influence language development include socio-economic status (Hart and Risley 1995) and the 
amount of reading materials in the home environment (Becher 1984). Vocabulary knowledge, 
fueled by language exposure, is a significant and constant predictor of overall reading 
comprehension (Yovanoff et al. 2005). 


Nippold (1998) summarized several emerging linguistic concepts that are particularly relevant 
when developing instruments for youth. Although the ability to order descriptors of magnitude 
such as “slightly” and “‘extremely” develops early, finer distinctions such as between “quite” and 
“decidedly” are not consistent even in adults. The understanding of analogies, metaphors, and 
similes also develops throughout childhood and adulthood; indeed, not all adults demonstrate 
mastery. Perceiving the difference between factive verbs (e.g., know, notice) which signal the 
truth of the clause that follows and non-factive verbs (e.g., think, believe) where the truth of the 
following clause is uncertain emerges relatively late, making it difficult for school-age children 
and adolescents to comprehend these differences. Likewise, understanding figurative meanings 
of idioms, proverbs, and humor begins to emerge in the early teen years with mastery typically 
reached in adulthood. Appropriately interpreting conjunctions (e.g., “but,” “if,” and “unless’’) are 
problematic for youth even in eighth grade (approximately ages 13-14). 


Reading development theory. Reading is a complex, language-based process, with lower-level 
skills such as decoding, vocabulary, phonetic awareness, and print knowledge preceding higher- 
level skills such as grammar and reading comprehension (Gough 1996; Share and Stanovich 
1995; Whitehurst and Lonigan 1998). 


With appropriate scaffolding, children can attend to text that is above their current reading level 
but within their zone of proximal development (Vygotsky 1987), especially if the content is 
relevant. A struggling reader may persevere through interesting text at a higher reading level, but 
abandon less engaging text that is theoretically at his or her level (Shiefele 1999). When the text 
is too far beyond their comprehension, it will be ignored (Halliday 1993). Although word 
recognition does not ensure good reading comprehension, it does appear to be a prerequisite for 
comprehension (Share and Stanovich 1995). Fluent readers have more available resources to 
concentrate on understanding. Larger vocabularies enable better comprehension and learning, 
and domain-specific knowledge affords readers even more fluency and vocabulary which further 
facilitates comprehension (Hirsch 2003). 


Although there are several available models describing how people learn to read, one of the more 
commonly accepted is Chall’s (1983) model of reading development. This theory posits that 
children progress through five stages as they begin to read: pre-reading (birth to approximately 
age 6); initial reading and decoding (approximately ages 6-7); confirmation and fluency 
(approximately ages 7-8); reading for learning (approximately ages 9 through 14); multiple 
viewpoints (approximately ages 14 through 18); and construction and reconstruction 
(approximately age 18 and up). Figure 2 provides additional detail about Chall’s (1983) model of 
reading, along with associated abilities and challenges. 
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Figure 2. Stages of Reading Development (Chall 1983) 


Reading development progresses through a series of continuous and overlapping stages (Chall 
1983). Some youth begin to transition to the confirmation and fluency phase as early as first 
grade (approximately ages 6-7), although the majority of fourth graders (approximately ages 9- 
10) have not mastered these skills (Treiman, Goswami, and Bruck 1990). As the curriculum 
shifts from “learning to read” to “reading to learn,” it is not uncommon for previously fluent 
readers to experience the “fourth grade slump” in their reading skills (Chall 1983). The slump 
may be due in part to the shift in reading assessments from decoding and fluency to 
comprehension (Hirsch 2003), but youth typically emerge with reading comprehension skills 
equivalent to their listening comprehension skills. This pattern holds until about 10th grade 
(approximately ages 15-16), when reading becomes more efficient for learning new material 
(Chall 1983). 


Social/Moral Development 


Collecting data is an inherently social encounter that is influenced by social norms and 
expectations. Children and youth differ in how they approach surveys and instruments according 
to their social and moral abilities. Theories about social and moral development are helpful for 
understanding how youth differ in their perceptions and interactions with others, as well as how 
they interpret and manage social and moral challenges. 


Social/Moral theories. Social reality is believed to consist of three broad domains: psychological, 
social, and moral (Turiel 1983a, 1983b). The psychological domain represents conceptual 
knowledge about personhood, such as the idea of a “self” and “others,” as well as psychological 
characteristics, such as feelings and personality traits. The social domain includes understanding 
of social rules and conventions and how people tend to act across situations. The moral domain 
involves justice and similar moral principles to demonstrate understanding of right from wrong. 
The two models of social development that are relevant to the survey response process are 
Selman’s (1980) model of perspective taking and Kohlberg’s (1976) model of moral reasoning. 
Perspective taking and moral reasoning skills develop sequentially, with no skipping of stages or 
regression once obtaining a particular stage (Walker 1982), and in parallel because gains in one 
domain support growth in the other (Colby et al. 1983). Figure 3 describes the stages of these 
two models. For the purpose of this paper, the focus is on the latter stages. 


Figure 3. Stages of Perspective-Taking (Selman 1980) and Moral Reasoning 
(Kohlberg 1976) 


Around ages 7 to 12, children become able to reflect on their own thoughts and actions 

from the viewpoint of another person, while realizing that others have this same ability (Selman 
Level 2). Children also understand that they may be unable to know others’ true thoughts and 
feelings, and are better able to recognize the potential for deception. Up until about 10 years of 
age, they view “morality” as following externally determined rules that do not require 
justification (Kohlberg Stage 1). Children begin to engage in a “tit for tat” morality where they 
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recognize the needs and interests of others but only as it relates to their own, and seek fairness in 
interactions to prevent retaliation (Kohlberg Stage 2). From about ages 10 to 15, youth can “step 
outside” of the two-person dynamic and consider both perspectives simultaneously (Selman 
Level 3). This process allows youth to consider not just the perspectives of themselves and 
others, but also to realize the implications of those perspectives for societal and legal contexts 
(Selman Level 4). These skills help youth understand how to successfully fill social roles 
(Kohlberg Stage 3). As adolescents continue to develop into adults, they begin making moral 
determinations from the perspective of society as a whole, rather than from personal interests. 
This reasoning seeks to preserve whichever social system is most important to the individual 
(Kohlberg Stage 4). Morals become the responsibility to preserve the social institutions above 
the individual (Kohlberg 1976). 


Integration of Developmental Stages 


Although cognitive, language/reading, and social/moral development occurs at different rates 
across children, there is considerable overlap across these domains. In particular, cognitive and 
language abilities are jointly involved with the perception, storage, and retrieval of information 
(Miller and Lenneberg 1978). Meanwhile, social skills inform comprehension and social norms 
and expectations that translate to the survey setting. For example, as youth begin to develop 
greater third-person perspective-taking skills (1.e., Selman’s Level 3), they are simultaneously 
experiencing a shift toward more sophisticated moral reasoning (Kohlberg’s Level 3). These 
changes occur during a similar timeframe as youth begin leaving the concrete operational stage 
of thinking and develop cognitive skills needed to think through problems hypothetically and in 
abstract terms in formal operations. This is representative of the drastic developmental shift 
many parents notice in their children around age 12. 


In viewing the stages of cognitive, language/reading, and social/moral skills together in Figure 4, 
it can be seen that at about the age of 12, most youth have entered the final stages of 
development. Because development in these domains stabilizes and prerequisite abilities are 
present, this age has important implications for instrument development because it represents an 
age at which instrument designers can begin to treat youth as adults. At the age of 12, working 
memory capacity is equivalent to adult levels, vocabulary is at a 5th grade reading level or 
higher, and youth are familiar with the social norms and perspective-taking skills expected of 
survey respondents. 


Figure 4. Cognitive, Reading, Social, and Moral Developmental Stages 

How Development Affects the Survey Response Process 

Cognitive, language/reading, and social/moral development provide the necessary foundational 
skills for children and youth to participate in instrument-based research. Designing instruments 
that result in reliable and valid data requires consideration beyond calculating grade-level 
equivalency of the text. To more fully understand measurement errors in children’s and youths’ 


responses, this section examines how the stages of the survey response model can be affected 
when cognitive, language/reading, and social/moral skills are not fully formed. 
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Comprehension 


The comprehension of a survey task is a function of cognitive, language/reading, and 
social/moral abilities. Because youth have underdeveloped cognitive skills, lower levels of 
language comprehension, and variability in their social and moral abilities (Bell 2007), 
comprehension is more difficult for youth than adults. 


Cognitive skills such as working memory help respondents remember and understand the context 
of survey questions, such as instructions, conditional information, and additional steps. Problems 
in the comprehension stage tend to occur because of the language used and how information is 
presented (Jenkins and Dillman 1997; Wright and Barnard 1975). Levine, Huberman, and 
Buckner (2002) combined ten questions with the common stem, “When you do mathematics in 
school, how often do you do the following?” into a matrix. By the time youths reached item 10 
(“Use a computer’), they had forgotten the context of “When you do mathematics in school,” 
and answered about their general computer use. Although increased working memory capacity 
should enable older youth to better remember contextual information, researchers should 
promote deeper processing by separating questions and incorporating contextual information into 
each question stem. Most children under eleven years old are in the Concrete Operations stage 
(Feldman 2004), and thus see the world in “black and white” terms, are less likely to understand 
abstract concepts, and have difficulty understanding hypothetical scenarios that are unrelated to 
any of their own personal experiences. This limits the content of questions, as well as the 
language that can be used. 


Language comprehension is another essential skill for the comprehension stage. As stated earlier, 
general population government surveys are designed to meet a 5th grade reading level (Biemer 
and Lyberg 2003). Since 2007, over 60% of children assessed in 4th and 8th grade have not met 
reading proficiency standards (U.S. Department of Education 2013). For youth, items should be 
written well below the intended grade level to ensure that youth can understand them. De Leeuw 
(2011) recommends that the text be written at a grade level that is two years lower than that of 
the target population. 


Defining grade-level writing is difficult. Reading statistics such as the Flesch-Kincaid Grade 
Level are based on the numbers of words, sentences, and syllables, and cannot account for 
sophisticated vocabulary or grammar. In the United States, reading proficiency standards differ 
across states, making it difficult to determine what on grade-level writing should look like. 
Although not widely adopted, the Common Core State Standards (National Governors 
Association Center for Best Practices, Council of Chief State School Officers 2010) define 
language arts and literacy skills for grades K-12, including specific rules for each grade level, to 
enable comparisons across states. These standards may be useful for instrument development 
because they define what youth should understand by the end of each grade. Elementary school 
teachers can also provide expert review of grade-level language and text. 


A large amount of text is burdensome for children, but there are several ways to simplify 
language. One way is to use more basic vocabulary. Younger children have smaller vocabularies 


and have not mastered advanced language concepts such as metaphors or humor (Nippold 1998). 
Abstract terms are problematic because they are generally too complicated for children under 12 
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years old. Although the use of examples can promote comprehension, youth may fixate on the 
provided examples and not generalize to the broader meaning of the term. Older youth 
understand abstract terms better. Another common pitfall is the use of survey-specific phrases 
such as “Very unimportant,” which are common in surveys for adults. This “surveyspeak” is 
awkward and may not be understood by all respondents (Harkness 2012). 


Other language concerns in children and youth are complex lexical and structural syntax 

and double negatives. Although a common approach to decrease reading difficulty is to shorten 
sentence length (De Leeuw 2011), this practice has the potential to instead increase the difficulty 
by removing important syntactical information (Davison and Kantor 1982). Language 
development markedly improves around age 12, enabling youth to understand more complex 
syntax. Children will have considerable trouble processing double negatives, which combine 
negatives in the question stem with negatives in the response options. Negatives can be 
incorporated into a question with the use of the word “not,” negative terms, negative prefixes, 
and negative suffixes. To reduce respondent burden, positively worded question stems, devoid of 
negatives, should be paired with a balanced set of response categories. 


Social and moral development influence the understanding of social norms and others’ 
perspectives. As youth become more socially aware, they may not ask for clarification out of a 
desire to appear knowledgeable or avoid embarrassment. Because youth may differ in their 
current world views and moral perspectives, they may interpret questions differently. For 
instance, children ages 10 and under would likely view a question about drug use in terms of 
following rules, whereas youth older than 10 might be able to adopt others’ perspectives and 
view issues from a societal or legal perspective. If respondents interpret a question differently 
due to their level of perspective-taking, comparability is compromised. 


Due to developing cognitive, language/reading, and social/moral structures, questions written for 
adults can easily confuse youth. Differences in these abilities may affect the validity of 
inferences. Researchers should employ a variety of measures to ensure that the target population 
can understand questions, including ensuring close proximity of contextual information, teacher 
reviews, and pretesting with targeted probes. 


Retrieval 


The retrieval stage can be especially difficult for younger children and youth due to their 
underdeveloped cognitive abilities. Measurement errors during the retrieval stage can be reduced 
by asking about events, attitudes, or behaviors that were encoded and are clearly represented in 
memory. Food consumption is notoriously difficult to report without a food diary—even for 
adults—because eating occurs with a high frequency and low rate of encoding. Retrieval tasks 
need to be simplified to match the target population’s cognitive skills and supported by 
encouraging appropriate retrieval strategies. 


Several strategies that support retrieval come from empirical studies with adults. One way to 


reduce cognitive load is to encourage the use of a retrieval strategy that matches how something 
is represented in memory (Conrad, Brown, and Cashman 1998). For instance, if an event can be 
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assumed to occur frequently and regularly, the events will blur together in a respondent’s 
memory and be difficult to count. Accordingly, researchers should encourage the use of a rate- 
based estimation strategy rather instead of an enumeration strategy. Time-related questions may 
be hard for young respondents due to an incomplete understanding of time. Although an 8-year- 
old child might be able to tell the time on a clock, he or she may have difficulty estimating the 
passage of time. Younger children may be more likely to use an estimation strategy during the 
retrieval stage. On the other hand, if an event is thought to be infrequent and irregular, 
researchers should promote an enumeration strategy (i.e., counting the actual number of 
occurrences) because they are more easily represented in memory (Menon 1993). Although short 
reference periods might help the recall of dates or frequency, instrument designers should test 
several reference periods to determine how far back respondents can recall with accuracy. 


Recall prompts are another method for stimulating retrieval. In asking about the total number of 
times the respondent left the classroom during a given day, Dirghangi (2014) provided several 
recall prompts: to use the bathroom, to get a drink of water, to go to the office or nurse, and so 
on. Although this method has been found to prevent under-reporting with 7- to 8-year-olds, it 
may also result in double-counting (Belli et al. 2000) or telescoping, which is counting an event 
that did not occur during the reference period. 


Finally, event history calendars (Belli 1998) may help support retrieval. Event history calendars 
help respondents remember by mapping memory cues such as seasons and salient times (e.g., “it 
was right before Christmas”). Because this method has been shown to improve the quality of 
autobiographical memory with adults (Belli 1998), it may be particularly useful for helping 
youth approximate time (Blair 2003). 


Retrieval is one of the most difficult stages of the survey response process, especially for 
younger children whose cognitive skills are not fully developed. In designing an instrument for 
youth, researchers can reduce errors during the retrieval stage by simplifying the difficulty of 
these questions and using one or more strategies to encourage recall, especially when time is 
involved. Retrieval tasks become easier when respondents’ working memory reaches full 
capacity at approximately twelve years of age (Cole and Loftus 1987; Kail 1990). 


Judgment 


During the judgment phase, respondents reflect on what is being asked and whether the 
information retrieved is adequate. This stage makes extensive use of working memory. Because 
working memory capacity is fully developed around age 12 (Cole and Loftus 1987; Kail 1990), 
older children will be better able than younger children to remember contextual information and 
complete multiple-step directions. Mathematical skills such as multiplication and division are 
needed to compute totals or averages. Measurement errors in younger respondents or those with 
weaker mathematics skills may be reduced with the use of two-part questions: first recalling 
information and then conducting secondary operations (e.g., averages). 


The ability to use metacognitive strategies such as self-questioning for comprehension can lead 
to the highest levels of understanding text (Haller, Childs, and Walberg 1988) and successful 
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reasoning through what has been read (Franks et al. 2013). The ability to engage in these types of 
metacognitive processes is not developed until a child reaches the formal operations stage at 
approximately 12 years of age. As youth proceed through adolescence, their ability to use 
metacognitive and self-regulation strategies improves (Huizinga, Dolan, and van der Molen 
2006). Because children and youth may not automatically evaluate their responses, it is possible 
to encourage these activities during the survey response process. Researchers could give 
respondents a metacognitive checklist for the survey response process, such as, “Do I understand 
what the question means?” or “How easy or hard was it for me to remember?” to indicate when 
they have problems with a question. Because children often use similar checklists in school for 
tasks such as writing stories (e.g., “Does each sentence begin with a capital letter?”’), a similar 
checklist could be used with children to trigger these evaluative processes. 


Responding 


The literature linking lower cognitive ability to measurement errors in older adults suggests that 
similar measurement errors in young respondents is due to incomplete cognitive development. 
Normal aging is associated with cognitive declines such as slower processing speed, reduced 
working memory capacity, and lower sensory function (Park 2000). Older adults with lower 
working memory are more susceptible to question wording, response format, and context effects 
(Knauper et al. 2007). Older adults also demonstrate more recency and primacy effects (Knauper 
et al. 2004; Schwarz and Knauper 2000). Lower cognitive skills may explain why younger 
children fail to recognize problematic aspects of questions, such as vague quantifiers or how to 
reconcile differences between the “true” answer and the offered response options (Borgers and 
Hox 2001). Children ages 7 to 10 may also be more susceptible to scale effects, scale numbering, 
and order effects than older youth (Fuchs 2005). 


Some language- and reading-related problems involve the response categories. Children may 
have difficulties anchoring and evaluating adverbs of magnitude (Nippold 1998), which can 
impair the validity of response options. Previous research has found that children ages 10 and 11 
prefer 5 options on a Likert scale (Rebok et al. 2001), and 6 or 4 options (Borgers, Hox, and 
Sikkel 2004). If younger children cannot fully differentiate between middle categories, however, 
it is better to use fewer response options that children can clearly understand. 


Researchers disagree on whether to include a “Don’t know” category for children and youth. 
Scott (1997) advocates for a “Don’t Know” option to be explicitly offered when surveying 
children to avoid guessing, but Holaday and Turner-Henson (1989) reported that children 
marked “Don’t know” responses not only to indicate true “Don’t Know” responses, but also lack 
of comprehension and refusals. Although multiple reasons may render the interpretation of a 
“Don’t Know” unclear, this category provides respondents with a way to indicate problematic 
questions. In a similar fashion, unmarked boxes in “mark-all-that-apply” questions also had 
multiple meanings such as insufficient processing, uncertainty, or refusal 

(Holaday & Turner-Henson 1989; Levine, Huberman, and Buckner 2002). For this reason, 
forced-choice question formats are preferred over “check-all-that-apply” because they encourage 
deeper processing and prevent primacy effects (Smyth, Christian, and Dillman 2008). 
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Understanding where youth are in terms of social development can help researchers understand 
youths’ responses. The perspective-taking and moral reasoning models suggest that youth under 
10 years of age should respect the authority of the test-giver and their ability to determine what 
rules need to be followed. Younger children believe adults have greater knowledge and power 
(Backett and Alexander 1991) and so may be anxious to please adults. Socially-desirable 
responding has been observed in children, both as underreporting of discouraged behaviors and 
overreporting of acceptable behaviors (c.f., Ogan, Karakus, and Kursun 2013; Schober et 
al.1992; Turner, Lessler, and Devore 1992). When teachers administer the instrument, younger 
children may try to avoid being “wrong” because they assume that adults know the “correct” 
response (Bell 2007). Acquiescence bias, or the tendency to agree, may also occur more 
frequently in children and youth due to their desire to please the interviewer (De Leeuw 2011; 
Maccoby and Maccoby 1954). Hence, the use of Agree-Disagree scales is strongly discouraged. 


Older youth are extremely concerned about privacy, so group administrations with respondents 
in close proximity may result in significantly fewer reports of sensitive behaviors 

(Beebe, Harrison, McRae, Anderson, and Rulkerson 1998). Older youth are also more likely to 
take a more nuanced approach to the morality of not following instructions or engaging in 
purposeful dishonesty, creating a “jokester group” of responders (Fan et al. 2006). Adolescents 
are less likely to perceive themselves as “bad” if they can justify their actions. Furthermore, as 
they develop the ability to recognize and use dual social realities, they may manipulate them to 
suit their purposes. 


Summary 


There are many ways in which measurement errors can occur throughout the response process. 
The integration of the survey response model with developmental theories yields a better 
understanding of when and how measurement errors manifest in children’s and youths’ 
responses. In particular, the findings connecting age-related abilities to measurement errors 
underscore the need for age-sensitive questionnaire design practices. 


Recommendations for Developmentally Appropriate Instrument Design 


Based on this review, a set of theoretical recommendations was developed to guide instrument 
design for children and youth. The recommendations build on existing guidelines for instrument 
design (e.g., Dillman, Smyth and Christian 2014; Fowler 1995; Schwarz 1996) with the goal of 
matching youths’ ability to complete questions, maintaining motivation, and minimizing 
developmentally-related errors throughout the response process. Table 1 provides both general 
recommendations for surveying youth between the ages of 9 and 18 (grades 4-12), and additional 
recommendations for youth ages 12 and younger, because 12-year-olds may not yet have reached 
the highest developmental stages. An overall recommendation for data collection with children 
under 12 is to provide up to twice as much time because younger youth have slower processing 
speeds and are less efficient readers. 


To minimize errors during the comprehension stage, encourage cooperation, and prevent 
boredom, all content should be relevant to the target population (Fan et al. 2006; Scott 1997). 
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Because youth may not be familiar with “survey-speak” (Harkness 2012), language and response 
options should reflect natural conversation. To reduce processing time, write in the form of 
questions rather than items. Aim for a reading level that is two grade levels below the target 
population’s grade, with the Sth grade reading level as a maximum. To help maintain the 
motivation of youth, question length should be limited, and as short and straightforward as 
possible for youth 12 and under. To encourage deeper processing, questions should not be 
combined into matrices or grids, especially when the question stem includes important 
contextual information. Due to the greater overlap in the social/moral stages, a given sample of 
12-year-olds may include youth with widely varying perspective-taking abilities. Therefore, 
questions requiring respondents to consider multiple perspectives simultaneously (e.g., “Some 
people believe ‘X,’ while others believe ‘Y.’ What do you believe?’’) should be limited to youth 
ages 16 and older. Hypothetical scenarios and vignettes that are unrelated to any personal 
experiences are also more likely to be difficult for youth under 12 (Inhelder and Piaget 1958). 
Because children in the concrete operational stage of cognitive development are likely to fixate 
on examples provided rather than generalize to the full meaning of a term, abstract terms should 
be avoided for youth aged 12 and under. Because electronic reading level calculations are based 
on syllable and word length, and not context or vocabulary knowledge, researchers are 
encouraged to enlist teachers to review language for a specific reading level. 


To minimize errors during to retrieval, it is best to ask about attitudes, events, or behaviors that 
are well represented in memory and are easy to recall. Youth respondents may also benefit from 
strategies to support autobiographical memory, such as memory cues in the question wording 
(e.g., “about how much” for estimation), or event history calendars (Belli 1998). Shorter 
reference periods may also be helpful, but should always be pretested with youth. 


To minimize errors during judgment and evaluation, there are two recommendations for youth 
ages 9 to 12. Questions asking for totals or averages require the respondents to retrieve and then 
conduct mathematical operations (e.g., addition, division), which requires considerable working 
memory. To reduce cognitive load, these multistep questions can simplified by breaking them up 
into separate questions. Because youth under 12 do not automatically use metacognitive and self- 
regulation strategies to evaluate their responses, a short checklist may encourage these behaviors 
and help them articulate problems with a question. 


Finally, to minimize errors during the reporting phase, there are three recommendations to 
increase the validity of responses. Because youth, especially children, are likely to agree with 
adults, Agree/Disagree scales should be avoided to prevent acquiescence bias. Fewer response 
options may also be beneficial for children ages 9 to 12, who are less able to differentiate 
between two or more similar options, particularly in the middle of the response scale. To reduce 
socially-desirability, avoid using people in positions of authority (e.g., parents, teachers) as 
interviewers, use self-administered modes and reassure youth that parents, peers, and other adults 
will never see their answers. When surveying older youth simultaneously in a group setting, 
provide a respectful distance from others. 


Researchers seeking to create instruments for youth must be prepared to consider developmental 


differences in the age range they are seeking to assess. These instrument design 
recommendations combine theory from several disciplines to inform instrument design practices 
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for youth ages 9 to 18. The recommendations aim to prevent and reduce measurement errors 
during the response process by providing guidelines for constructing instruments to match the 
target population’s cognitive, language/reading, and social/moral abilities. 


Conclusions 


Very little methodological research has been conducted on children and youth. In designing 
instruments for children and adolescents, researchers have much to learn from educational and 
developmental psychologists about developmentally-appropriate adaptations. This review 
considers young respondents’ capacities to participate in research and the measurement errors 
occurring throughout the response process that can be attributed to development. 


Prior to this review, previous research examined how cognitive development may affect the 
survey response process in youth. This review builds on previous work by additionally 
examining the language/reading and social/moral abilities that are also needed to participate in 
survey research. Although it was not clear when these skills become sufficient to conduct survey 
research, this review suggests three major thresholds. Children aged six and younger form one 
group that have wide-ranging and insufficient abilities, and so research with this age group may 
be best served via proxies. Children between the ages of seven and 11 form a second group that 
is heterogeneous with respect to their cognitive, language, social, and moral skills, but are better 
able to answer questions than younger children. Instrument design pitfalls for this group include 
complicated vocabulary, syntax, and grammar; estimating the passage of time; computing 
averages; and advanced perspective-taking. The third group consists of adolescents aged 12 and 
older, who have most skills needed to complete instruments with minimal errors related to 
developmental concerns. This group should be able to answer nearly every type of question, 
although they may lack advanced social skills such as perspective-taking. 

Perhaps of most interest to researchers is that age 12 seems to be a transitional threshold at which 
youth begin to be capable of completing instruments designed for adults. Not only have most 12- 
year-olds entered the final stages of development, but most can also read at a 5th grade reading 
level and have the working memory capacity of an adult. Yet, it is important to recognize that 
12-year-olds should not be treated as adult respondents in every respect—content must be 
appropriate and relevant. Because not all 12-year-olds may not have reached the highest stages, 
the recommendations provided here group 12-year-olds with youth aged 9 to 11. 


It is hoped that these theoretical recommendations will inspire additional research. The 
thresholds described above are theoretically-based, so research is needed to empirically test 
them. For instance, much remains to be learned about how different levels of these skills affect 
responses to certain question types. Research in these areas is necessary to improve measurement 
with children and youth. 
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Table 1. 


Instrument Design Recommendations for Children and Youth 


Response Stage 


Overall 


General Recommendations for 
Youth 


Recommendations for Ages 12 
and Under 

Provide plenty of time; do not rush 
respondents 


Comprehension 


- Ensure that content is relevant to 
youth 
- Use a conversational tone 
- Write questions (not items) 
- Limit question length and 
complexity 
- Reading level should be two 
grades 
below target (maximum 5th 
grade level) 
- Do not combine questions into 
matrices 
- Limit questions about multiple 
perspectives 


- Questions should be as short and 
simple as 

possible 
- Hypothetical scenarios must be 
relatable 
- Avoid abstract terms 


Retrieval 


- Ask about things that were 
encoded 

and are clearly represented in 
memory 
- Encourage strategies to support 
memory 
- Pretest reference periods 


Judgement 


- Simplify multi-step questions 
- Consider providing a 
megacognitive 

checklist 


Reporting 


- Avoid agreement scales 

- Emphasize privacy and 
confidentiality 

- Provide a respectful distance from 
others 


- Use two to four response 
categories 


Note. General recommendations are for youth ages 9 - 18. 12-year-olds are grouped with youth ages 9 to 11 because 
they may not yet have progressed to higher developmental stages. 


